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Abstract. The 321 , hexagon-avoiding (321 -hex) permutations were introduced and 
studied by Billey and Warrington in M as a class of elements of S„ whose Kazhdan- 
Lusztig and Poincare polynomials and the singular loci of whose Schubert varieties 
have certain fairly simple and explicit descriptions. This paper provides a 7-term 
linear recurrence relation leading to an explicit enumeration of the 321-hex permu- 
tations. A complete description of the corresponding generating tree is obtained as a 
by-product of enumeration techniques used in the paper, including Schensted's 321- 
subsequences decomposition, a 5-parameter generating function and the symmetries 
of the octagonal patterns avoided by the 321-hex permutations. 
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1. Introduction 

We start by describing the 321 , hexagon-avoiding permutations in two ways; first in 
the context of pattern-avoidance, which is the viewpoint we will be exploiting in our 
enumeration, and then in the context of reduced expressions, which explains the intro- 
duction of the term 321, hexagon-avoiding (for brevity, 321-hex). Finally, we explain 
briefly the connection with Kazhdan-Lusztig polynomials in and how this motivates 
the work on the present paper. 

1.1. Pattern— avoidance. From the first viewpoint, we consider permutations in S n 
as bijections w : [n] — * [n], and write them in one-line notation as the image of w, 
[wi,W2, ■ ■ ■ ,w n ]. For n < 10 we suppress the commas without causing confusion. 



Key words and phrases. 321-hexagon-avoiding permutations, forbidden subsequences, heaps, linear 
recurrence, Kazhdan-Lusztig polynomials. 
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Definition 1. Let v £ Sk and w £ S n for some k < n. We say that w contains v if there 
is a sequence 1 < i\ < ■ ■ ■ < < n such that the sequences w' = [w^ , Wi 2 , ■ ■ ■ , Wi k ] and 
[vi,V2, ... ,Vk] obey the same pairwise relations, i.e. Wi j < Wi m exactly when Vj < v m . 
In such a case, we write w' ~ v. If w does not contain v then we say that w avoids v. 
We denote by S n (v) the set of all v-avoiding permutations of length n. 

For example, the permutation u = (52687431) avoids (2413) but does not avoid (3142) 
because of its subsequence (5283). For a classification of forbidden subsequences up to 
length 7, we direct the reader to @, 1, 0, HI, 0, 0, Hi- 



Definition 2. The 321-hex permutations are those permutations which simultane- 
ously avoid each of the following five patterns: 

[321], Pi = [46718235], P 2 = [46781235], P 3 = [56718234], P 4 = [56781234]. 

We denote by T the set of the four length-8 (octagonal) permutations P±, P3, 
In order to make sense of the above definition, consider the following equivalent, but 
perhaps more insightful, reformulation in terms of matrices. 

Definition 3. Let w S S n . The permutation matrix M w is the n x n matrix having a 
1 in position (i,Wi) for 1 < i < n, and elsewhere. (To keep the resemblance with the 
"shape" of w, we coordinatize M w from the bottom left corner.) Given two permutation 
matrices M and N, we say that M avoids N if no submatrix of M is identical to N. 




Figure 1. Octagonal Patterns 

A permutation matrix M of size n is simply a transversal of an n x n matrix. Clearly, 
w £ S n contains v S if and only if M w contains M v as a submatrix. Under this 
reformulation, Fig. |l| presents the four octagonal patterns {Pi} which must be avoided 
by the 321-hex permutations. The fifth pattern P does not come from a permutation 
because it is not a transversal. Yet, P is a union of the four previous permutation 
patterns, and it can be easily checked that a permutation matrix M w avoids all p's if 
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and only if no 8 x 8 permutation submatrix of M w can be completely "covered" by P. 
Thus, by abuse of notation, we can say that the 321-hex permutations are defined as 
the permutations avoiding both [321] and P. 

Our enumeration makes use of this last interpretation, exploiting the symmetries in 
the set y of octagonal patterns plus a convenient structural representation of all 321- 
avoiding permutations. The close relations between the four octagonal permutations 
are even more clearly revealed from a group-theoretical viewpoint when examining 
their reduced expressions. Below, we briefly review the construction of the heaps of 
321-avoiding permutations and their relation to the octagonal patterns P^; for more 
details, see 0. The remainder of the introduction, except for its conclusion, can be 
skipped by the reader who is interested only in the pattern-avoidance interpretation. 

1.2. Heaps of 321— permutations. The permutation group S n can be regarded as 
generated by the set of the adjacent transpositions {sj}^ , where Sj = (i, i + 1) in cyclic 
notation. In this presentation, the generators Sj and Sj commute if \i — j\ > 1; else 
SiSi + iSi = Si + iSiSi + i. An expression is any product of generators. A reduced expression 
(w) for w G S n is a shortest-possible expression yielding w. (It is well-known that the 
number of generators in (w) equals the number of inversions in w.) For example, the 
octagonal pattern Pi = [46718235] has a reduced expression 

(Pi) = S3S 2 SiS5S4S3S2S 6 S5S4S3S 7 S 6 S5- 

Reduced expressions for the other avoided octagonal patterns are (P2) = (Pi) • S4, 
(P3) = S4 • (Pi) and (P4) = S4 • (Pi) • S4. These expressions can be easily verified by 
considering the dashed lines in P in Fig. |l[ The study of reduced expressions is a major 
subject of the representation theory of S n . 

After work of Viennot [jig], the 321-avoiding permutations w can be represented 
by special ranked posets called heaps. The elements of Heap(w) are identified with 
the transpositions {fij } in a fixed reduced expression (w) for w. By Billey-Jockusch- 
Stanley ||, the 321-avoiding permutations are those in which no reduced expression 
contains a substring of the form SiSi±iSi; and by Tits [15| , all reduced expressions for 
a 321-avoiding permutation w are equivalent up to moves SiSj — > SjSi for \i — j\ > 1. 
Thus, the set of elements in Heap(w) is independent of the choice of (w). 



1 2 4 3 5 3 2 1 3 2 1 




1 2 3 4 5 1 2 3 1 2 3 



Figure 2. String diagrams for S3 G S5, S2S1S2 G 53 and S1S2S1 G .S3 

We now describe the rank function of Heap(w7), along with a Hasse diagram for 
the poset by embedding its elements in the integer lattice. One way to define and 
visualize this embedding is via string diagrams. To form a string diagram of w G S n , 
write the row of numbers [wi, W2, w n ] above the row [1, 2, n], thus mimicking the 
two-line notation for a permutation. Connect each number i on the bottom line to the 
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corresponding number % on the top line, drawing a "string" which may change direction, 
but which at all times is running either due north, northwest or northeast. Strings may 
cross and recross, but do not run over top of one another, nor do they stray beyond 
the rectangular bounds formed by the two rows of numbers. For example, S{ £ S n can 
be realized as the crossing of two strings, as shown in Fig. ^a. In general, two or more 
adjacent transpositions can be applied simultaneously, provided they commute, i.e. Sj 
and Sj can occur on the same horizontal level of a string diagram unless \i — j\ = 1. 

Thus, the crossings of a string diagram can be identified with the Sj's, labelled by 
the column (from 1 to n — 1) in which they occur, and can be seen to form a poset 
in the obvious way. The linear extensions of this partial order are the expressions for 
w defined above. If the string diagram has the smallest possible number of crossings, 
then it is is minimal and the linear extension is a reduced expression for w. 

A short braid is a configuration obtained by applying the non-commuting transpo- 
sitions Si and Sj + i in the following order: SjSj+iSj or Sj+iSjSj+i (cf. Fig. ||b-||c). The 
string diagram for a braid shows crossings at three of the four points of a small diamond, 
omitting either the eastern or the western point. As mentioned earlier, a permutation is 
321-avoiding exactly when its minimal string diagrams avoid such configurations; such 
permutations are therefore also called short-braid-avoiding in the literature of Coxeter 
groups. It is now clear how to embed canonically the poset of string crossings in a min- 
imal string diagram of w into the integer lattice. The resulting Hasse diagram is the 
heap of w, Heap(w). It is independent of the choice of (w) as long as w is 321-avoiding. 

The heap of the special reduced expression (Pi) for the octagonal pattern Pi resem- 
bles a hexagon: Heap(Pi) has horizontal and vertical symmetries, with respectively 
2,3,4,3,2 lattice points on its five ranks (see Fig. ||a-||b). The string diagram of P4 
features one extra point on top and one extra point on bottom, corresponding to the 
crossings of the strings 1 and 8, and 4 and 5 (see Fig. |3|c). The string diagrams for P2 
and P3 have either the top or the bottom extra crossing. In all cases, Heap(Pj) contains 
the hexagonal Heap(Pi). 




Figure 3. String diag rams for Pi and P4, and Heap(Pi) where (Pi) — rii=if^ 

Furthermore, it can be shown that the (321, T)-avoiding permutations are exactly 
those 321-avoiding permutations whose minimal string diagrams avoid the hexagonal 
string diagram of Pi. This justifies the descriptive term 321, hexagon-avoiding and 
yields the following alternative description (cf. Q): 

Definition 2'. A 321-hex permutation is a permutation whose reduced expressions do 
not contain a substring of the form SjSj±\Sj for all j > 1, and do not contain any of 
the translates n*;=i s ik+j 

of Heap(Pi) for all j > 0. 
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1.3. Kazhdan Lusztig polynomials. We now turn our attention to the Kazhdan- 
Lusztig polynomials. For definitions and a detailed introduction, we direct the reader 
to ||,||. In short, the Hecke algebra "X of a finite Weyl group W (such as S n ) is an 
algebra over Q(y / g), with basis {T w } w€ ]y, relations for all generators s £ W: 

T S T W = T sw if l(sw) >l(w), 

T s 2 = ( 9 -i)r a + gr a , 

and well-defined inverses due to the presence of q . Moreover, the involution of 
Q(y/q) sending ^fq ^ 1/y 7 ? extends to an involution i of Ji. The Kazhdan-Lusztig 
polynomials arise in search of a new basis {C^,} for "K of t-invariant elements which are 
linear combinations of "lower-terms" T x for x < w under the Bruhat-Chevalley order: 
x < w if every reduced expression for w contains a subexpression for x. 

Theorem 1 (Kazhdan-Lusztig). For any w £ W , "K has a unique i-invariant element 
C w = g'W/^^Pj^, where the degrees of the polynomials P x , w (q) £ %[q] « r e at 
most \{l{w) — l(x) — 1) if x < w, and P w>w = 1, P x , w = if x ^Cw. 

The Kazhdan-Lusztig polynomials P X] w are of fundamental importance in Lie The- 
ory. It has been proven that their coefficients are non-negative for Weyl groups (see 
Kazhdan-Lusztig [10|, Brylinski-Kashiwara Q), but this question remains open for 



arbitrary Coxeter groups. Neither the degrees nor the coefficients of P XjW are readily 
computable; indeed, only partial results have been derived in certain cases (see [||] for 
detailed references.) 

1.4. 321 hex permutations in Kazhdan Lusztig polynomials. In ||], Billey and 
Warrington introduce the 321-hex permutations w G W = S n , derive simple combi- 
natorial formulas for their Kazhdan-Lusztig and Poincare polynomials, and a simple 
method for determining the singular loci of their Schubert varieties X w . Concretely, let 
(w) = Sj x ■ • ■ Si r be a reduced expression for w, and let d(a) denote the defect statistic 
of a mask a on (w) whose product is a fixed x £ S n . (For definitions of these and other 
related concepts, see j|, |[|.) 

Theorem 2 (Billey- Warrington). A permutation w £ S n is 321 -hexagon avoiding if 
and only if one of the following equivalent conditions is satisfied: 

(a) The Kazhdan-Lusztig polynomial P x , w = q d ^ for x < w. 

(b) The Poincare polynomial for the full intersection cohomology group of X w equals 
(1+ q) li - w \ 

(c) The Kazhdan-Lusztig basis element C' w = C' Si ■ ■ ■ C' s . . 

(d) The Bott-Samelson resolution Y of X w is small. 

(e) IH*(X W )~H*(Y). 

1.5. Enumeration of 321 hex permutations. As Theorem || shows, the 321-hex 
permutations have properties that make certain algebraic and combinatorial computa- 
tions easier to perform compared to arbitrary permutations. Another interesting aspect 
of the 321-hex permutations was examined earlier in the introduction: they are one of 
the few families that are known to be describable both in terms of pattern-avoidance 
and heap-avoidance. Yet, until now, there was no known recursive, exact or other 
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closed form for the number of 321-hex permutations. For instance, it would be useful 
to know to how many permutations Theorem || applies; how the number of 321-hex 
permutations changes asymptotically, and how it compares to sizes of other well-known 
sets of permutations, such as 5^(321), which is enumerated by the Catalan numbers. 

Answers to all these questions are obtained in the present paper, where we find a 
7-term linear recursive relation and derive from it an explicit exact formula. With this 
formula at hand, answering any enumeration questions about the 321-hex permutations 
becomes a matter of simple observation and calculation. 

Definition 4. Let "K n denote the set of all 321-hex permutations in S n , and let a n = 
\Ji n \ be the number of such permutations. 

Theorem 3. The sequence a n satisfies the following recursive relation for all n > 6: 
a n = 6a n _i - lla„_2 + 9a n _ 3 - 4a n _ 4 - 4a„_ 5 + a„_ 6 . 

Of the six roots of the corresponding characteristic polynomial, four are real: Ri for 
i = 1,2,3,4, and two are complex conjugates: R$ = Rq. This implies the same 
description of the six coefficients below: q S M for i = 1, 2, 3, 4, and C5 = cq £ C. 

Corollary 1. The number of the 321 -hexagon avoiding permutations of length n equals 

Cl R\ + c 2 R r 2 l + c 3 R% + c A R n A + c 5 R% + ^R^, 
where the roots and coefficients are rounded off below to 5 digits after the decimal point: 



R^ 


! -0.49890 


Cl P 


a 0.00164 


R 2 - 


! 0.21989 


c 2 P 


a 0.13776 


R 3 - 


! 1.95627 


C3 s 


a 0.57156 


R4 ~ 


! 3.43526 


C 4 P 


a 0.24149 


R 5 * 


! 0.44375 - 1.07682i 


C5 p 


s 0.02378 + 0.00080i 



For further discussion of this and other results, we refer the reader to Sections ||-[|. 
The proof of Theorem || follows several steps. First, we describe the nodes in the 
generating tree of !H n by using Schensted's algorithm for 321-avoiding permutations 
and by introducing 5 parameters for the generating function h n (x,k,l,m). We next 
observe that this function depends on fewer parameters, yielding therefore relatively 
few distinct values. We organize these values in five sequences, a n , (3 n , 7„, S n and e n . 
Using the intrinsic symmetries of the set 7 of octagonal patterns, we deduce recursive 
relations expressing each sequence in terms of a n . The latter turns out to be the 
number we are looking for, |CK n |. Finally, putting together all information about the 
generating function and the five sequences results in the desired formula. 

2. The generating tree 

2.1. What is a generating tree? We turn now to the development of a recurrence 
for the 321-hex permutations. A standard tool in the enumeration of restricted per- 
mutations is the generating tree T introduced in 0]. Begin with an infinite tree whose 
nodes on level n are identified with the permutations in S n . The node w is a child of 
w = [wi, . . . , Wj-i, • • • , w n ] where the omitted value is Wj = n. Looking at this 



EXPLICIT ENUMERATION OF 321,HEXAGON-AVOIDING PERMUTATIONS 



7 



from the point of view of the parent, we can form all the children of w G S n by inserting 
the element n + 1 into each of the n + 1 sites of uu. 

Then, for a given set £ of forbidden subsequences, prune the tree by deleting all 
nodes containing any of the forbidden subsequences. What remains is still connected 
because if w does not contain any forbidden subsequence then clearly w does not either. 
For any node on level n of this pruned tree T(£), we call a site in the corresponding 
permutation active if inserting n + 1 at that site yields a node of the tree; conversely 
an inactive site is one where the insertion of n + 1 creates one or more of the forbidden 
subsequences in S. 

To give a complete description of a generating tree, we need to associate to each 
node an appropriate label, and then describe a succession rule for deriving the labels 
attached to the set of children of each node. For instance, we might characterize the 
original tree T generating all permutations as having a root labelled (2) and a succession 
rule (n) — > (n + l) n . In this instance, the label can be interpreted as revealing directly 
how many children each node has in the generating tree. 

In the particular instance of the 321-hex permutations, it will turn out that we need 
a label containing four integers. Although this is more complicated than the single 
integer of our motivating example, it is nevertheless a major progress to reduce the 
amount of information recorded at a node from a full permutation to a label of any 
bounded size. In particular, it is possible to apply the succession-rule recursively to 
determine the entire downward structure of any node given only its label, regardless of 
whether that node is on a level corresponding to permutations on four symbols, or four 
thousand. 

2.2. Schensted algorithm for S' n (321) and active regions. Let w be any 321- 
hexagon-avoiding permutation on n symbols. We divide the elements wi,W2, ■ ■ ■ ,w n 
into two categories: the set of right -to-left minima (including w n ), and the rest. Adapt- 



ing the terminology of Schensted [11], we refer to these as the first basic subsequence 
and second basic subsequence of w, and denote them by "Bi(w) and 2*2 (w), respectively. 
For example, when w = Pi, l Bi(w) = [4, 6, 7, 8] and S 2 (w) = [1, 2, 3, 5]. Note that both 
Hi(w) and ^(w) decrease from right to left, the former - by construction, and the 
latter - because w £ 5 n (321). Now let K = w^, L = Wi 2 and M = Wi 3 be the three 
largest (i.e. rightmost) elements in £2(1^) (K < L < M), with each one set to zero if 
the corresponding element does not exist. 

Since M is the largest (i.e. rightmost) element in £2(10), it follows that every element 
to the right of M belongs to r S>\{w). There are x := n — 13 elements in this region, which 
we will call the active region. If M = because T>2(w) is empty, then we consider the 
entire permutation to be the active region. Let k be the number of elements in the 
active region which are larger than K; as the elements in the active region decrease 
from right to left, these k elements are w n _fc+ij ... ,w n . Similarly, I be the number of 
elements in the active region which are larger than L, and m be the number of such 
elements larger than M (cf. Fig. |l|.) 

Now assign the label (x,k,l,m) to w. By construction, x > k > I > m. Let the 
X -elements of w be those that are counted in x, and similarly define the K-, L-, and 
M-elements. Further, let the X\K -elements of w be the set of X-elements minus the 
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set of if-elements, and similarly define the K\L— and L\M-elements. For example, 
the X -elements in P\ are 2, 3, 5, and k = I = m = 0. 

2.3. The succession rule for 321 hex permutations. All children of w must avoid 
the subsequence [321]. This restriction by itself renders inactive all sites to the left of 
M, but none of the sites in the active region. Therefore when considering the children 
of w and their labels, we need only consider insertions taking place in the active region 
to the right of M. Thus, if we insert n+1 into a site in the active region with j elements 
to its right, it is easy to verify that the resulting permutation will have label: 



(1) 



' (x + 1, k + 1, 1 + 1, m + 1) if j = 

if0<j<m 
{j,j,rn,0) \lm<j<l 
e(j, I, m, 0) if I < j < x 



Furthermore, the number of w's children can be computed as follows. Set 

T := mm(k + 2, max(k + 1,1 + 2)). 

Then the node w has S + 1 children, corresponding to the S + 1 rightmost insertion 
sites, where 



(2) 



S = T ifT<x-2, 
S = x if T > x - 2. 



This allows a more compact and complete succession rule for the labels of all S + 1 
children of w: 



(3) (x,k,l,m) 



(x + I, k + l,l + l,m + l), 
[i, min(z, /), min(i, m), 0) for i = 1, S. 

We shall not use directly any of the above formulas (HI)-® in our calculations, so 
we leave their verification to the reader, who will find this easier after mastering the 
material in the rest of the paper. The importance of the above discussion is that it 
completely describes the structure of the generating tree T(321, 7), and hence explains 
in principle why the enumeration in this paper works. Why the resulting final formula 
for the 321-hex permutations is so simple - a linear recursive relation with constant 
coefficients - is a completely different matter and can be explained only by the structure 
of the forbidden set J of four octagonal patterns, as we shall see later. 

2.4. The generating function for T(321, T). Let h n (x, k, I, m) and 3i ra (x, k, I, m) be 
the number, resp. the set, of 321-hex permutations in S n labelled (x, k, I, m). According 
to this definition, h n (0, 0, 0, 0) is not defined since no permutation is labelled (0, 0, 0, 0). 
For convenience, denote by h n (0, 0,0,0) and 3f n (0, 0,0,0) the number and the set of 
all 321-hex permutations in S n which end in their largest element: w n = n. It is worth 
noting that "K n {x, x, x, m) for x > m corresponds to permutations w in which either L 
is smaller than the final "tail" of w, or L does not exist and ^(w) = {M}. 

Naively, there are 0(n 5 ) enumerations to be performed for h n (x,k,l,m). However, 
many have the same answers; indeed, only 5n different values of h n+ \(x, k, I, m) appear: 
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n 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


a n 


1 


2 


5 


14 


42 


132 


429 


1426 


4806 


16329 


55740 


190787 


Pn 








1 


4 


14 


48 


165 


568 


1954 


6717 


23082 


79307 


In 











1 


5 


20 


75 


271 


957 


3337 


11559 


39896 


Sn 














1 


6 


25 


93 


333 


1172 


4083 


14137 


e n 

















1 


5 


19 


68 


240 


839 


2911 



Lemma 1. The operation of deleting all K-elements in a 321 -hex permutation pro- 
vides a bijection dx '■ Ji n (x,k,l,m) —> < K n _} t (x — A;, 0,0,0). Hence, for all n, x, k,l,m 
we have h n (x, k, I, m) = h n -k(x — k, 0, 0, 0). 



Proof: Let w G S n (321). The if-elements of w lie in "Bi(w) and are part of the final 
increasing "tail" of w (since they are to the right of M). Thus, if a K-element Wi 
were part of an octagonal pattern Pj in w, then Wi would lie in Hi(Pj). But then Pj, 
and hence w, would contain at least 3 elements larger than and to the left of wf. this 
contradicts the definition of a K-element, for which only L and M are larger than and 
to the left of it. 





« 


1 y 



l-m 




1 


a 

k-l 


• 






o x-k 



delete 
K-elements 





















• 






o x-k 



Figure 4. Lemma 



The above discussion shows that JC-elements cannot participate in octagonal pat- 
terns in w, and hence they can be deleted without losing any relevant 321-hex infor- 
mation about w. (Of course, we have to rescale down appropriately M and L of w to 
arrive at a permutation of smaller size.) The resulting map 

dx '■ $Cn{%, k, I, m)— >IK n _fe(z — k, 0, 0, 0) 

is bijective: to obtain w G "K n (x,k,l,m) from its image w = dx{w) G 3f n _fc(x — 
k, 0, 0, 0), insert the necessary number of M, L, if-elements into w and increase appro- 
priately L and M to fit their definitions in w. This procedure works because we can 
identify M as the largest element in w (after the deletion of all iT-elements in w), and 
then identify L as the second largest element of w, which will be necessarily to the left 
of M . Then insertion of the appropriate M, L, /T-elements at the end of w requires 
rescaling M and L only (not K), and hence transforms the 321-Schensted decomposi- 
tion of w into that of w: whether L was in ¥>i(w) or in ¥>2(w) does not prevent L from 
becoming an element of H>i(w) after applying d K . □ 
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2.5. Octagonal conditions. Consider a family 3~ of permutations in "K n which are 
described by certain configuration conditions imposed on their basic subsequence de- 
composition Si U I>2. For example, ?{ n (0, 0, 0, 0) is such a family defined by the condi- 
tion w n = n € r S>x{w). Let a be an element of the permutations in 3~ which is identified 
uniquely in each w £ 3" by the configuration description of 3". 

Definition 5. We say that a is pattern-free if its deletion in each jd£ J (and appro- 
priate rescaling of w) results in a numerically equivalent family 3" of permutations in 
3f n ; i.e. d a : 3"^ 3" with |3~| = |3~|. For w £ 3", denote by u; the image d a (w) G 3". 

For example, in IK n (0, 0, 0, 0), the largest element a = n is identified by being in the 
last position in each w, and clearly it is pattern-free (no octagonal pattern Pj has 8 in 
its last position): 



d n :3{ n (0,0,0,0) ^% 



n-l- 



Establishing pattern-free elements and identifying the image set 7 is the basis of the 
enumeration of "K n . The following technical lemma summarizes the pattern-free situ- 
ations which will be used later in the proof of Theorem ||[ 



a x _k be the X\K -elements of w £ 3", and 
The set 7 imposes the following octagonal 



Lemma 2. Let 3" be a family in !K n , a\, 
H be the fourth largest element in H>2(w 
conditions on 3~: 

(PI) All K -elements are pattern-free. 

(P2) If w n = n inJ , or w n = n — 1 in J , then w n is pattern-free. 
(P3) If M = n and x < 2 in 3", then M is pattern-free. 

Assume now that x > 3 and k = I = m = in 3 r . 

(P4) If H < a2 in J (H may not exist in w,) then M,a2, ...,a x are pattern-free 
(P5) If a2 < H < a% in 3, then L, M, 03, a x are pattern-free. 
(P6) If H > a% (forcing x = 3) in 3", then M is pattern-free. 




(P2)-(P3) (P4)-(P6) 

Figure 5. Octagonal Conditions 

Proof: (PI) follows from the proof of Lemma [[]. (P2) says that if all w € 3" end in 
their largest element n (or in their second largest element n — 1), then this last element 
is pattern-free (cf. Fig. ||); (P3) says that if M happens to be the largest element 
in all w £ $ and is followed by at most two X-elements, then M is pattern-free (cf. 
Fig. g-§.) Both (P2) and (P3) follow from the facts: 

• Each octagonal pattern Pi has an empty 3x3 upper-right corner (see Fig. ||a.) 
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For (P2), reinserting n or n — 1 in the last position in w £ "Kn-i does not create 
321-patterns. Similarly, for (P3), reinserting M = n into w £ "K n — 1 as the first 
(rightmost) element in ®2(itf) does not create 321-patterns (since the tail of w 
after M is increasing as part of H>\{w).) 



For (P4), cf. Case 1 in Fig. 1C. Start by deleting the crosses of the elements in the 
bottom row and in the rightmost column of each Pj; this leaves the permutation matrix 
M[345612], which decomposes into a 4 x 4 block Z4 in the upper left corner and the 
fixed 2x2 block I2 in lower left corner (corresponding to the original p^ = 2 and pj = 3, 
see Fig. |5|b.) 

• The fact that I4 is to the left of and higher than I2 proves that a2,---,a x are 
pattern-free: for them, only M,L,K can fit in such an I4. In detail, if one of 
CL2, ...,a x participates in an octagonal pattern Pj, then without loss of generality 
we may assume that a x is the rightmost element of Pj. This forces M,L,K to 
participate in Pj too as the only elements larger than a x . Now M, being the largest 
element of Pj, forces at least three elements after it to participate in Pj; one is 
a x , one could be a\, hence the third one must be among 02, a x -l- But none of 
these elements can fit into an I2 as in Fig. || because only L, M, N are larger and 
to the left of them. Further, reinsertion of 02, a x into w as the largest elements 
of Hi(w) cannot create 321-patterns. Thus, 02,03, ...,a x are indeed pattern-free. 

• After deleting 02, ...,a x , the largest element M lands in second to last position in 
w, and by (P3), it is pattern-free. 



For (P5), cf. Case 2 in Fig. 10. The reasoning here is similar to the case for (P4). 

• First note that there can be no element ao of w between L and M, or else 
[H,K,L,a ,M,ai,a2,a 3 ] ~ P x . 

• If one of as, a x participates in a pattern Pj, this forces M, L, K to participate 
too, and in order not to run into contradiction with the Z4 x I2 argument above, 
we must assume that a\ and 02 are also in Pj, and only one among as, a x is in 
Pj. Then the "1" in Pj will have to be between the two largest elements L and 
M, which was ruled earlier. Thus, 03, ...,a x are pattern-free. 

• Deletion of 03, ...,a x leaves M G ¥>2(w) in third to last position, so by (P3), M 
is pattern-free. But L is the largest element in w = d,M{w), with only a\ and 02 
after it, so by (P3) again, L is pattern-free. 



For (P6), cf. Case 3 in Fig. 11. Again note that there can be no element ao of w between 
L and M, or else [H, K, L, Oq,M, ai, 02, 03] ~ P3. If M participates in a pattern Pj, then 
it forces 01,02,03 also to participate. Since the "1" in Pj cannot come from between 
L and Af, we conclude that L does not participate in Pj. But then we can replace M 
by L and argue that there is a pattern Pj in w = dM(w), a contradiction. Hence M is 
pattern-free. □ 



3. Relations among a, f3, 7, 5 and e 

Lemma [l] shows that h n (x,k,l,m) does not depend on / or m, but rather on the 
differences n — k and x — k. We shall see further that there are only 5 ranges for x — k, 
that completely determine the values of h: each of x — k = 0, 1, 2, 3 and x — k > 4 
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corresponds to exactly one of the 5 sequences listed earlier, and denned as follows: 



h n (x, 


X 


-0,/, 


m) 


= K- 


^+0(0,0,0,0) =: 


«n 


-x-1 




On 


'■= h n+ i 


(0,0,0,0) ' 


h n (x, 


X 


-1,1, 


m) 


= K- 


. x+1 (1,0, 0,0) =: 


Pn- 


-x+0 




0n 


'■= h n+ i 


(1,0,0,0) 


< h n (x. 


X 


-2,1, 


m) 


= K- 


^+2(2,0,0,0) =: 


ln- 


-x+1 


> 44> < 


In 


'■= h n+ i 


(2,0,0,0) > 


h n (x, 


X 


-3,1, 


m) 


= K- 


^ +3 (3, 0,0,0) =: 




-x+2 




s n 


= h n+ \ 


(3,0,0,0) 


h n (x, 


X 


x, I 


m) 


= K- 


- x+ - x {x, 0,0,0) = 


■ e n - 


-x+3 t 






— hn+x- 


-3(5,0,0,0) 



where x > 4. All of these definitions are justified by Lemma [j], except for the definition 
of e n , which depends only on n but not on x and whose explanation will be given later. 
In our search for relations between these sequences, it will be useful to define each 
sequence via an alternative description in the terminology of the families 3 discussed 
m Section @. 

3.1. The sequence a. By definition, a n = h n+ i(0, 0, 0, 0). M = n + 1 is the largest 
and the rightmost element of any w 6 !K n +i(0, 0, 0, 0), and hence it is pattern-free by 
(P2) (see Fig. ^.) Deleting M results in a family of 321-hex permutations of length n 
without any further restrictions, i.e. we have a bijection 

(0,0,0,0)^^ n = |J M n (x,k,l,m). 

x,k,l,m 

This justifies the alternative description of a n , stated in the Introduction: 
(4) a„ = # {321-hex permutations in S* n } = \K n \. 




Figure 6 . Alternative Description of a. 



3.2. The sequence j3. By definition, (3 n = h n+ i(l, 0, 0, 0). Here M = n + 1 is second 
from right to left in w £ Jf n +i(l, 0, 0, 0), and by (P3) it is pattern-free (see Fig. [?].) 
Thus, deleting M imposes only one extra condition: in the new 321-hex permutation 
w £ "Kn, the rightmost element in "Bi(w) is a i^\L-element or lower, i.e. the largest 
two elements (the original L = n and K = n — 1) belong to H>2{w). This justifies the 
alternative description: 

(5) = #{w e Ji n | {n,n - 1} C ^(w)} for n > 3. 

In order to find a recursive description of /3 n , note that each w € "K n either ends in 
n or n — 1, or both n and n — 1 belong to ^(u 1 ): 

(6) a n = #{w £"K n \n last} + #{w £^K n \n-l last} +/?„. 

» , ' > , 

«n-l Oin-l 
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Figure 7. Alternative Description of /3, 



By (P2), if n or n — 1 is the last element in w £ 'Kn, then it is pattern-free, so deleting 
it results in a bijection: 

d n : {w G "Kn | n last} — > 3f n -i and d n _i : {u> G !K n | n — 1 last} — > 5C n -i- 

This justifies the use of a n _i = |CK n -i| in @. Consequently, we have the recursive 
relation j3 n = a n — 2a n _i for n > 3. □ 

3.3. The sequence 7. By definition, 7„ = /i n+ i(2, 0, 0, 0). Here M = n + 1 is third 
from right to left in w £ J{ n+ i(2, 0, 0, 0), and by (P3) it is pattern-free (see Fig. ||.) 
Deleting M imposes the following extra conditions: in the image w G the first 
two elements in iBi(io) are X\L-elements or lower, i.e. the largest two elements (the 
original L = n and K = n — 1) belong to H>2{w) and there are at least 2 numbers a/ier 
L = n. This justifies the alternative description for n > 4: 

(7) 7n = #{^ e I {w s = n, w t = n - 1} C S 2 (t»), * < s < n - 2}. 




Figure 8. j n 

In order to find a recursive description of 7„, note that each w counted in (3 n (i.e. 
{n, n — 1} C B2(w)), falls into one of the following subcases: there is exactly one 
element after n (hence in !Bi(io)), or there are at least two elements after n (hence in 
BiH): 



(8) p n = #{w G "K n I {Wn-x = n, n - 1} C £ 2 H} +7n- 

V v ' 

«n-l - «n-2 

The underbraced set in @ is depicted in the LHS of Fig. [| By (P3), deletion of L = n 
results in the numerically equivalent set S in the RHS of Fig. U. The permutations in S 
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can be described as having their largest element K = n — 1 E e B>2{w). On the other hand, 
"Kn-i breaks into two disjoint groups: group A consists of the permutations having the 
largest element n — 1 in last position, and group B is the set S: 3i n -i = A\JS. Finally, 
note that d n -\ : A ^ "K n -2, so that |5| = a n _i — a n _2- This justifies the underbrace 
notation in and implies the formulas: j n = (3 n — (a n -\ — a n -2) = a n — 3a n _i + a n _2 
for n > 4. □ 



L=n 



1 

n-1 




• 


• 



delete K=n4 



Figure 9. Calculation of j n 



3.4. The sequence e. By definition, e n = h n+x s(x, 0, 0, 0) for x > 4. Let a\,...,a x 
be the X\_£T-elements of w G JC„_|_ x _3(x, 0, 0, 0), where x > 4. There are two cases to 
consider (see Fig. |i~0| .) 

Case 1. Except for M,L,K, all other elements of ^(w) are smaller than 02- Our 
drawing shows the next element H E 22(10) s.t. H < ai (H may not exist in w.) By 
(P4), a2, «3, «4, a x and M are pattern-free. After deletion, the remaining configura- 
tion in !H n _3 is identical to the alternative description of (3 n -3- 




Figure 10. Calculation of e 
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Case 2. After M,L,K, the fourth element H of r S>2{w) is between a<i and 0,3: a>2 < 
H < 03. Recall from (P5) that there can be no element ao of w between the vertical 
lines of L and M; or else, ao < a%, and hence [H, K, L, ao, M, a%, 0,2, 03] ~ Pi- Further, 
03, 04, ■■■,a x , M and L, are pattern-free. After deletion, the remaining configuration in 
!K„_3 is identical to the alternative description of j n -3- 

Note that the case H > 03 is not allowable, or else [H, K, L, M, a\, az, 03, 04] ~ P2 
or ~ P4 (when < H < ov H > 04, respectively.) 

Incidentally, the above discussion shows that h n+x ^s(x, 0,0,0) does not depend on 
x as long as x > 4, and hence justifies the definition of e n . We conclude that e n = 
Pn-i + 7n-3 = 2a n _ 3 - 5a„_ 4 + a n _ 5 for n > 7. □ 

3.5. The sequence 5. By definition, 5 n = h n+ \(3, 0, 0, 0). As above, denote by 
01, 02,03 the X\iT-elements of w S 5£ n +i(3, 0, 0, 0). There are three cases to consider 
(see Fig. 0.) 
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Case 1. Except for M,L,K, all other elements of l E>2(w) are smaller than 02. Our 
drawing shows the next element H 6 r S>2{w) s.t. H < 0,2 (H may not exist in w.) By 
(P4), 02,03 and M are pattern-free. The remaining configuration in "K n -2 is identical 
to the alternative description of (3 n -2- 

Case 2. After M,L,K, the fourth element H of r E>2(w) is between 02 and 03: 02 < 
H < C13. By (P5), there can be no element ao of w between the vertical lines of L and 
M; or else, ao < a±, ao G B2(w>), and hence [H, K, L, ao, M, ai, 0,2, 03] ~ Pi. Further, 
03, M and L are pattern-free. After deletion, the remaining configuration in "K n -2 is 
identical to the alternative description of 7„_2- 

Case 3. In contrast to the discussion of e, in the case of 5 it is possible to have H > 03 
as long as there is no element ao of w between the vertical lines of L and M; otherwise, 
[H, K, L, ao, M, ai, 02, 03] ~ P3. By (P6), the largest element M = n + 1 is pattern- 
free. After its deletion, the remaining configuration in < K n -i is identical to the original 
description of S n -±. 

We conclude that S n = /3„_ 2 + 7n-2 + 6 n -i = ^n+i + S n -i for n > 6. □ 

We summarize the results in this Section in 

Lemma 3. The sequences a, (3, 7, 5 and e satisfy the following recursive relations: 

Pn = a n - 2a n -i for n > 3; 

7n = a n - 3a„_i + a n _ 2 for n > 4; 

S n = e n+ i + 5 n -i for n > 5; 

e n = 2a„_ 3 - 5a„_ 4 + a n _ 5 /or n > 6. 

4. Enumeration of 321-hex permutations 

We are now in a position to combine the recurrence formulas for a, (3, 7, 5 and e into 
a single recurrence for a. We first use the interpretation of a n as \K n \ and expand this 
in terms of the individual values of h n (x, k, I, m). Thus, for a fixed n 

a n = ^2 h n (x,k,l,m), 

where the sum is taken over n>x>/c>/>m>0, x 7^ 0. Next, we break the sum 
into five separate sums depending on the value of x — k; each such sum corresponds to 
the definitions of a, (3, 7, S and e, respectively. Note that the sum for a (where x = k) 
requires two extra special cases for x = k = n — 1 and x = k = n; in both cases 
h n (x, k, I, m) = 1. 

a n = l + (n— 1) + ^ h n (x,x,l,m) + y~]h n (x,x - 1,1, m) 

n — 2 > x > n>x 
x>;>m>0 x-l>i>m>0 

+ ^^/i n (x, x — 2, /, m) + x — 3, /, m) + h n (x,k,l,m). 



x-2>l>m>0 



x-3>l>m>0 



x-i>k>l>m>0 
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In the next step, we replace the /i n 's by the appropriate values of a,(3,j,5 and e. 
The case of a requires some care as we must observe the condition that not all the 
numerical parameters x,k,l,m can be simultaneously equal; indeed, this only happens 
in the special case x = k = l = m = n, which was broken out from the main sum 
earlier. 

=> a n = n+ a n _ x _i + } y /3 n - x + } y 7re-x+i + ^ 5 n - x+2 + e n -a-+3- 

n — 2>a;,x>m n>x n~>x n>x n>x 

x>l>m>0 x-l>i>ra>0 x-2>l>m>0 x-3>l>m>0 x-i>k>l>m>0 

We replace the indices k, I and m by binomial coefficients: 

n— 1 re— 1 re— 1 re— 1 re— 1 

«re = n + ^ (rr)-i) + E + E ( n+ "^ + E o^ 1- *)* + E 

i=l j=3 i=4 i=5 i=6 

Finally, we use the recurrences from Sect. || to obtain a summation in terms of a alone: 

re— 1 re— 1 re— 1 

OL n = n + Y^ (("+2 1 " 1 )- 1 ) a i +E ( U+ 2' l )( a i - 2 «i-l) +E (" + 2" ! )(«i - 3 «i-l + <*t-2) 
i=l i=3 i=4 

re— 1 «— 4 

+ E ( n+ 2~ ,; ) ( 2tti - 2 ~ 3ai " 3 ~ 2 E a 3 ~ 4 ° 2 + ai ) + (V) ff 5 
i=5 j=3 

re— 1 

+ E C l+ 3~ l )( 2 «i-3 - 5ai_ 4 + ai_ 5 ). 
i=6 

We simplify this large expression into the following full-history, linear recurrence rela- 
tion with cubic polynomial coefficients, valid for n > 6: 

re— 4 

a n = 2a n _i + 3a n - 2 + Q{k)a n ~k + 

fe=3 

where Q(jfe) = -\k 3 + ^k 2 - ^ + 9 and R(n) = -^-n 3 + 85n 2 - ^f-n + 704. The 
polynomial R(n) arises from three special cases for the polynomial coefficients of a^, a 2 
and ql\. Since degQ(n) = degi?(n) = 3, we need 4 successive history eliminations of 
the form a n — a n -\; combined with the 3 initial terms a n ,a n -i and a n -2, this produces 
the desired order-six constant -coefficient linear recurrence for all n > 6: 

(9) a n = 6a n _i - lla n _2 + 9a n _ 3 - 4a n _ 4 - 4a n _ 5 + a„_ 6 . 

Consequently, the number of the 321-hexagon avoiding permutations of length n is 
given by the formula: 



(10) c x R n x + c 2 R% + c 3 R% + c 4 R2 + c 5 R% + c 5 R 5 n , 

where the roots and coefficients are listed in the Introduction. This completes the proof 
of Theorem 0. □ 
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The sixth-degree characteristic polynomial of our recurrence relation (^) is irre- 
ducible over Q and has Galois group 56, as calculated by Maple. This means that 
there are no further algebraic relations among the roots Ri in (|l0|), and thus we cannot 
hope for any better closed-form results. 

However, our numerical approximations of the roots and coefficients can yield exact 
values for the number of 321-hex permutations for any fixed length n, exploiting the fact 
that the value being approximated is known to be an integer. Furthermore, since the 
two roots of modulus less than 1 make such small contributions, they can be dropped 
and the following formula is exact for all n: 



(11) a n = [c 3 B% + c A R n 4 + c 5 B% + cjRl>], 

where the braces denote rounding to the nearest integer. 

5. Extensions and Further Discussion 

The study of the octagonal patterns Pi in the present paper was motivated by their 
apperance in the representation theory of S n via heap-avoidance and thus Kazhdan- 
Lusztig polynomials and Schubert varieties. We refer to the enumeration of 5" n (321, !P) 
as the "8 x 8 case". From a purely combinatorial viewpoint, it is natural to ask what 
happens in the analogous smaller 6x6 and 4x4 cases whose generalized patterns are 



depicted in Fig. 12 




Figure 12. 6x6 and 4x4 Patterns 

To obtain these cases, for each i = 1,2,3,4 shorten by one the lengths of both Si(Pj) 
and ¥>2(Pi) by removing a chosen fixed point appearing in all Pi's. Thus, define 

IP 6 = {[351624], [356124], [451623], [456123]} and 

7 4 = {[2143], [3142], [2413], [3412]} 

to be the families of avoided octagonal patterns in the 6x6 and the 4x4 cases, 
respectively. Both of these cases lead again to linear recursive relations with constant 
coefficients. The proofs below follow closely the method described in the 8x8 case, so 
we leave the details for verification to the reader. 

5.1. The 6 x 6-case. 

Theorem 4. Let = S' n (321, y 6 ), and a n = |!K^|. Then {a n } satisfies a 6-term 
linear recursive relation with constant coefficients: 

(12) a n+ i = 4a n - 4a n _i + 3a„_ 2 + a n _ 3 - a n _ 4 for all n > 1. 
Consequently, for all n > 1, 

(13) a n = Cl R^ + c 2 R% + c 3 R% + c 4 R2 + c^RJ 
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where the roots and coefficients are rounded off below to 5 digits after the decimal point: 

Ri ps -0.49569 a ps 0.63205 

R 2 ps 0.51154 c 2 « 0.53110 

#3 ~ 3.03090 c 3 ps 0.50154 

ps 0.47662 - 1.03635i c 4 ps -0.19482 + 0.11092i 

The degree-5 characteristic polynomial of the recurrence is irreducible over Q and has 
the largest possible Galois group, S$, as calculated by Maple. We can again drop the 
small roots R\ and R 2 , rounding off the remainder to the nearest integer: 

(14) a n = [c 3 R% + c A Rl + clRj] for all n > 1. 

The first values of a n are: 1, 2, 5, 14, 42, 128, 389, 1179, 3572, 10825, 32810, 99446. 

PROOF(Theorem ||) : We modify the discussion in the proof for the 8x8 case. Define the 
generating function h n (x,l, m) with one fewer parameter, thus taking into account only 
the elements M and L of w S !K n . As in Lemma [l], the L-elements are pattern-free, 
and therefore h n (x, I, m) = h n _i(x — I, 0, 0). 

Next, define the sequences a n = /i n+ i(0, 0, 0), /3 n +i = h n (l,0,0), 7 n = /i n+ i(2, 0, 0) 
and S n = h n+x _2(x, 0, 0) for x > 3. Note that a n = h n+ i(0, 0, 0) = \"K n \ since a n is the 
number of w € in which the largest element n + 1 is at the end and hence it is 

pattern-free. Further, the relations among the sequences are: 

fi n = a n - a n _i for n > 2; 

7„ = a n - 2 + /?n-2 + 7n-l for n > 3; 
<5 n = a n _ 3 + /3 n _ 3 for n > 4. 

Now we are ready to express everything in terms of a n = Y^ x i m hn( x i I, Tn): 
a n = 1 + h n (x, x, m) + ^ h n (x, x — l,m) + y] /i n (x, a; - 2, m) + ^ /i n (^, ^, w). 

Here the "1" counts the identity permutation: no M-element in w; in the first sum L 
is smaller than all X -elements (or L does not exist); in the second sum L is larger than 
exactly one X -element; in the third sum L is larger than exactly two X-elements; and 
in the forth sum L is larger than three or more X-elements and hence x — 3 > I. 

n— 1 n— 3 n— 4 n— 5 

a n = 1 + lon-l-i + + ^"-'-l + + !)7n-I-l + Yl (W»-J-l 

n Ti—2 n— 3 n— 4 

=> «n+l ~ On = ^ «n-i + ^ /^n-i + ^ 7n-/ + £G + 1)<5„-, 
1=1 1=0 1=0 1=0 

n—3 

=> a«+2 - 2a n+ i + a n = a n + /9 n+ i + 7„+i + <5 n _z+i 

1=0 

=> Oti+3 - 3a n+2 + 2a n+ i = {(3 n +2 ~ Pn+l) + {ln+2 ~ 7n+l) + #n+2- 

Conveniently, each summand on the RHS can be expressed in terms of a, including 

7n+2 - 7n+i = a n + j3 n = 2a n - a n -i: 

=> ati+3 = 4a n+2 - 4a n+ i + 3a n + a n _i - a n _ 2 . □ 
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5.2. The 4x4 case. 

Theorem 5. Let = 5„(321, ? 4 ) ; and a n = (IK 4 ). Then 

(15) a n = (n- l) 2 + 1 for all n > 1. 

In particular, {a n } satisfies the J^-term linear recursive relation 

(16) a n+ i = 3a n - 3a„_i + a n _ 2 /or all n > 1. 

Note that in contrast to the 6x6 and 8x8 cases, the characteristic polynomial here 
factors (completely) over Q: (x — l) 3 . 

Proof: As expected, we define the generating function h n (x, m) with only two param- 
eters, keeping track only ofMinwG 3f 4 . Obviously, the M-elements are pattern-free, 
so h n (x,m) = h n - m (x - m,0). Further, letting f3 n = h n+1 (l,0) and 7 n = h n+x -i(x, 0) 
for x > 2, one quickly discovers that (5 n = /3 n _i + 1, so j3 n = n + 1 for n > 3, and 
j n = n — 1 for n > 1. Express Q n = ^t n (a;, m) as the sum 

a n = l + '^2 h n (x, x- 1) + y~] /i n (x, m). 

As before, here the "1" counts the identity permutation: no M-element in iu; in the 
first sum M is larger than exactly one X-element; and in the third sum M is larger 
than two or more X-elements and hence x — 2 > m. 

n—2 n— 3 

=^ a n = 1 + /?n-m-l + 7n-m-l 
m=0 m=0 

=>• a„+i - a n = /3 n + 7„ =>■ a n+2 - 2a n+i + a n = 2 
=>■ a n+3 - 3a n+2 + 3a n+ i + a n = for all n > 1. □ 



5.3. Further Discussion. Until now, there were relatively few known examples of sets 
of permutations whose avoidance led to linear, polynomial, or exponential formulas (see 



12, [Lq].) After the successful enumeration of the 4 x 4, 6 x 6 and the 321-hex 8x8 



cases, it is tempting to generalize the recursive sequence method of this paper to the 
corresponding larger sets of 2k x 2k patterns. At this point, it is not surprising to con- 
jecture that all these families yield linear recursive relations with constant coefficients. 
In fact, when the result of the present paper was publicized, Herbert Wilf requested 
that many more examples of such "linear" families be found. These and other related 
questions will be answered positively in a forthcoming paper by Stankova-Frenkel. 
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